Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented as a FILE_SERVER variable passed to data loaders (as an environment variable), allowing them to query an asset server.
For instance, a bash data loader
mags.txt.sh
can query another file (say,quakes.json
) by calling,and a JavaScript data loader will call:
In preview, when
quakes.json
is updated,mags.txt
gets updated. Ifquakes.json
is in fact generated by a data loaderquakes.json.sh
, then touching that script live-updatesmags.txt
. (The interpreter used by any of the data loaders is inconsequent: python can talk to typescript, and vice-versa.)We track the dependency "graph" on-disk, by associating to any file (say
filename.json
) generated by a data loader a filesrc/.observablehq/cache/filename.json__dependencies
that is a simple list of the paths that it requested against itsFILE_SERVER
.TODO:
error on circular dependencies, maybe?closes #332
To test this easily within the preview server:
cp test/input/build/chain/chain*.* docs/
then open http://127.0.0.1:3000/chain
You can replace
3
in chain-source.json.sh by$RANDOM
if you want to check that the source data loader runs only once for its two dependents.Old description
We must use the same file server for browser preview and machine calls (i.e. chained data loaders), because concurrent requests on a same data loader must be joined.
But the paths are different. The data loader for
caller.csv
receives a$SERVER
environment variable equal tohttp://127.0.0.1:3000/_chain/caller.csv::
, and might retrieve a dependency by concatenating that variable and the file path it needs, e.g. calling$SERVER/dependency.zip
.This should make it possible to derive the dependency graph, at least after we run the data loaders (not sure how to maintain state when we restart a server and we have a cache). Also, if the file is not found, we send an empty 404 instead of the decorated page intended for the browser; this makes it a bit more foolproof (tip: use
curl -f
to fail on 404).The current server information is saved as a global (in
process.env
—should it beglobalThis
?) for now. It feels a bit wrong, but at the same time it really is a global state.